What the bug is: At kimik2.5_fp4_mi355x.sh:47, the newly added expert-parallel check uses POSIX single-bracket syntax. POSIX [ ] requires both operands of -gt to be non-empty integers; when EP_SIZE is empty or unset, bash prints '[: : integer expression expected' to stderr and exits the condition with a non-zero status, causing the else branch (no expert parallel) to be taken.

The specific code path: The new YAML search-space entries include configs without an ep field (e.g., { tp: 8, conc-start: 4, conc-end: 64 } and { tp: 4, conc-start: 4, conc-end: 64 }). For those runs, EP_SIZE is either not exported to the shell environment or is exported as empty. The check on line 47 fires and produces the stderr noise on every such invocation.

Why existing code does not prevent it: EP_SIZE is intentionally absent from the check_env_vars call at the top of the script, because it is genuinely optional - only the { tp: 4, ep: 4, ... } entry needs it. There is no set -e in the script, so execution continues to the else branch correctly, but the spurious diagnostic is still emitted.

Addressing the refutation: One verifier argues the CI sweep generator (generate_sweep_configs.py line 677) always supplies EP_SIZE=1 as a default when ep is absent, meaning the error never occurs in automated CI runs. This is plausible - if that default is reliably enforced by the pipeline, the bash error is never triggered in practice. However, the script is also invocable manually for local testing or ad-hoc runs where EP_SIZE may not be set, and the defensive pattern costs nothing to adopt.

Step-by-step proof (manual invocation):

1. Set only the required env vars: MODEL=... TP=8 CONC=4 ISL=1024 OSL=1024 MAX_MODEL_LEN=... etc.
2. Leave EP_SIZE unset.
3. Run: bash kimik2.5_fp4_mi355x.sh
4. Bash reaches line 47: [ "" -gt 1 ] emits 'bash: [: : integer expression expected' to stderr.
5. Execution continues with EP=" " (no expert parallel) - correct behavior, but the spurious error was still emitted.

How to fix: Replace line 47 with: if [ "${EP_SIZE:-0}" -gt 1 ] (default to 0 when unset) or equivalently: if (( ${EP_SIZE:-0} > 1 )). Both handle empty or unset EP_SIZE silently without changing any observable behavior.

Bug Analysis

The newly added MEC firmware version check at benchmarks/single_node/kimik2.5_fp4_mi355x.sh:39-42 reads:

version=$(rocm-smi --showfw | grep MEC | head -n 1 | awk '{print $NF}')
if [[ "$version" == "" || $version -lt 177 ]]; then
  export HSA_NO_SCRATCH_RECLAIM=1
fi

How the Bug Manifests

The condition uses short-circuit OR with two branches: "$version" == "" handles the empty case, and $version -lt 177 handles the numeric comparison. A third case is unhandled: when version is a non-empty, non-numeric string such as N/A or unknown — common sentinels in ROCm tooling.

Inside bash [[ ]], the -lt operator performs arithmetic comparison. When the left-hand operand is a non-integer string, bash either silently coerces it to 0 or produces an error and returns false. The verifiers confirmed that for non-integer strings the comparison returns false in practice.

The Specific Code Path

1. rocm-smi --showfw outputs a firmware table; grep MEC | head -n 1 | awk '{print $NF}' extracts the last field of the first MEC line.
2. If that field is N/A, version is set to N/A.
3. [[ "N/A" == "" ]] evaluates to false (not empty).
4. [[ N/A -lt 177 ]] — bash arithmetic on a non-integer returns false.
5. Short-circuit OR: false || false → false — the if body is skipped.
6. HSA_NO_SCRATCH_RECLAIM is never exported.

Why Existing Code Does Not Prevent It

The empty-string guard correctly covers the case where awk produces no output. It does not cover non-empty, non-numeric values. There is no type check or regex validation before using -lt on $version.

Impact

On systems where rocm-smi --showfw returns a non-numeric firmware version field, HSA_NO_SCRATCH_RECLAIM=1 will silently not be set. Per the script comment, this flag is needed on MEC firmware older than version 177 to prevent RCCL scratch-memory reclaim crashes. A machine with old firmware but a non-numeric version string in rocm-smi output would be left in the crash-prone configuration without any warning. The practical risk is low since rocm-smi output is normally numeric on production hardware, but the violated fail-safe assumption is a real defect.

Step-by-Step Proof

1. rocm-smi --showfw on a specific machine outputs a MEC line whose last field is N/A.
2. version is assigned the string N/A.
3. [[ "N/A" == "" ]] evaluates to false.
4. [[ N/A -lt 177 ]] — bash arithmetic comparison with non-integer — evaluates to false.
5. Entire OR condition is false; if body is not entered.
6. HSA_NO_SCRATCH_RECLAIM remains unset, leaving potential for crash on older firmware.

Recommended Fix

if [[ "$version" == "" || ! "$version" =~ ^[0-9]+$ || $version -lt 177 ]]; then
  export HSA_NO_SCRATCH_RECLAIM=1
fi

The added ! "$version" =~ ^[0-9]+$ clause ensures that any non-numeric (indeterminate) version string causes the safety flag to be set, correctly implementing the fail-safe default.

The firmware version check is intended to protect against crashes on older MEC firmware by setting HSA_NO_SCRATCH_RECLAIM=1. The check reads:

version=`rocm-smi --showfw | grep MEC | head -n 1 | awk '{print $NF}'`
if [[ "$version" == "" || $version -lt 177 ]]; then
  export HSA_NO_SCRATCH_RECLAIM=1
fi

How the bug manifests: If rocm-smi --showfw outputs a non-numeric last field for the MEC line — for example N/A, unknown, or a version string with a non-numeric suffix — the empty-string guard passes (version is not empty), but the second operand $version -lt 177 attempts a bash integer arithmetic comparison on a non-integer. Inside [[ ]], this silently returns false (bash evaluates it as an error/false rather than raising a fatal error). Due to short-circuit OR semantics, the overall condition is false, so HSA_NO_SCRATCH_RECLAIM is never exported.

Why existing code doesn't prevent it: The empty string check "$version" == "" only handles the case where awk produces no output. It does not guard against non-integer strings like N/A that awk could reasonably extract as the last field of rocm-smi output. Bash integer comparison $var -lt N inside [[ ]] does NOT raise an error that propagates — it simply evaluates to false, making the entire OR condition false.

Concrete proof: Suppose rocm-smi --showfw outputs a line like ... MEC ... N/A. After awk '{print $NF}', version=N/A. Then:

1. [[ "N/A" == "" ]] → false (short-circuit does NOT skip second condition)
2. [[ N/A -lt 177 ]] → bash arithmetic error, condition evaluates to false
3. Overall [[ false || false ]] → false
4. HSA_NO_SCRATCH_RECLAIM=1 is NOT set

The machine may then experience the RCCL memory reclaim crashes the flag was meant to prevent.

Impact: When firmware version is indeterminate (non-numeric), the workaround is silently skipped. The safe/fail-safe behavior should be to apply the workaround when the version cannot be confirmed to be ≥ 177.

Fix: Add a numeric guard as the middle condition:

if [[ "$version" == "" || ! "$version" =~ ^[0-9]+$ || $version -lt 177 ]]; then
  export HSA_NO_SCRATCH_RECLAIM=1
fi

This ensures: empty string → set flag, non-numeric string → set flag, numeric < 177 → set flag, numeric ≥ 177 → do not set flag.